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Abstract 
A small proportion of students do not benefit sufficiently from standard intervention protocols, 
and require more intensive, individualized instruction. Data-Based Instruction (DBI) has a strong 
evidence base for addressing students’ intensive academic needs, yet it is not widely 
implemented. In this study, we explored the usability and feasibility of a professional 
development system to support teachers’ use of DBI in writing. Data analyzed using a mixed 
methods design revealed that teachers perceived supports such as coaching as facilitators of DBI 
implementation, whereas access to materials and external factors such as time conflicts presented 
challenges. Teachers made statistically significant growth from pre- to posttest on a measure of 
DBI knowledge and skills, implemented DBI components with fidelity, and reported that time 
spent on DBI activities decreased each week, supporting its usability and feasibility. Findings 
suggest that DBI is usable and feasible when teachers are provided ongoing professional 
development supports. 


Keywords: data-based instruction (DBI), usability, feasibility 
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Usability and Feasibility of Data-Based Instruction for Students with Intensive Writing Needs 

Many children who experience academic difficulties, including those at risk for or 
identified as having disabilities, benefit from high-quality, research-based instruction that 
follows a standard treatment protocol; however, a small proportion require more intensive, 
individualized intervention (Wanzek & Vaughn, 2009). Indeed, intensive, individualized 
instruction was intended to be a cornerstone of the Individuals with Disabilities Education Act 
(2004), yet such instruction is often absent from current special education practice (Fuchs, Fuchs, 
& Stecker, 2010; Fuchs, McMaster, Fuchs, & Al Otaiba, 2013; Lemons, Al Otaiba, Conway, & 
Mellado De La Cruz, 2016; Zigmond, 2001). Unfortunately, many students whose individual 
learning needs go unmet experience substantial and long-term negative consequences, including 
school dropout and under- or unemployment (Wagner, Kutash, Duchnowski, Epstein, & Sumi, 
2005), prompting calls to prepare educators who are better equipped to improve outcomes for 
students with the most significant learning needs (e.g., National Center on Intensive Intervention, 
n.d.). 

Data-Based Instruction 

One approach to effectively individualize and intensify academic instruction is Data- 
Based Instruction (DBI; Fuchs et al., 2010). DBI (originally termed “Data-Based Program 
Modification” by Deno and Mirkin [1977] and also referred to as data-based individualization, 
e.g., National Center on Intensive Intervention) is a hypothesis-driven, empirical approach to 
individualization (Deno & Mirkin, 1977) that entails an ongoing cycle of assessment and 
instruction delivered in addition to—or instead of—core instruction and small group intervention 
over an extended period of time (Danielson & Rosenquist, 2014). More specifically, DBI is a 


framework that includes using reliable, valid assessment data (typically from curriculum-based 
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measurement [CBM], which serves as a global indicator of overall performance in an academic 
domain [Fuchs & Deno, 1991]) to (a) establish a student’s present level of performance and set 
long-term goals, (b) monitor student response to high-quality instruction, (c) apply systematic 
decision rules to determine if a student is on track to meet goals, (d) generate hypotheses about 
ways to individualize instruction when needed, and (e) test the effects of instructional changes 
(cf. Jung, McMaster, Kunkel, Shin, & Stecker, 2018). 

A strong evidence base supports the use of DBI to improve outcomes for students with 
intensive needs in reading, mathematics, and spelling, particularly if expert or peer consultation 
or technology is in place to support teachers’ DBI implementation (Stecker, Fuchs, & Fuchs, 
2005). In fact, a recent meta-analysis of DBI literature revealed an overall effect size of g =.37— 
.38 (Jung et al., 2018) for DBI implementation. In this meta-analysis, effects of DBI varied 
depending on the type of CBM tasks used (with teacher-generated CBM tasks yielding larger 
effects than researcher-generated tasks), frequency of CBM administration (with more frequent 
administration yielding larger effects), and type and frequency of supports provided to teachers 
(with more frequent support that included collaborative problem solving and individual support 
to teachers yielding larger effects). Studies conducted in the area of spelling or writing yielded 
larger effects (g = .47) than those conducted in reading or math. Further, most studies 
emphasized teachers’ use of CBM data to make instructional changes but did not provide 
extensive professional development (PD) related to actual instructional options. One study that 
placed strong emphasis on instruction (focused on writing) as part of the DBI process (Jung, 
McMaster, & delMas, 2017) yielded a relatively strong effect (g = .63). 

However, despite these research syntheses supporting the use of DBI to improve student 


outcomes, it is not widely implemented in practice (Fuchs et al., 2010; Fuchs et al., 2013; 
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Lemons et al., 2016). Researchers have offered numerous explanations for this gap between 
research and practice, including that teachers are inadequately prepared to individualize 
instruction; inclusive practices are currently over-emphasized rather than focusing on intensive, 
individualized intervention; and practical challenges such as large caseloads and paperwork 
burdens detract from valuable instructional time (Fuchs et al., 2013). 

In light of these challenges, our research team recently set out to develop a PD system 
comprised of four full days of face-to-face content, fidelity checks, and ongoing coaching to 
support teachers’ use of DBI with children with intensive early writing needs. We focused on 
early writing given the importance of writing to children’s literacy development and success in 
school and later life (Biancarosa & Snow, 2004; Graham & Perin, 2007), and the lack of a focus 
on writing in the DBI literature. Though writing is a key focus of our work, the primary aim of 
our project was to develop supports to ensure that DBI was usable and feasible for teachers to 
implement in their classrooms. 

In this paper, we focus on the extent to which these supports facilitated teachers’ use of 
DBI, in an effort to develop, research, and share DBI practices that are supported by evidence, 
and also usable and feasible for teachers to implement. In addition, as implementation of DBI 
becomes more prominent, we hope that our approach will aide other researchers who are 
interested in supporting teachers’ practices. 

Facilitators and Barriers to Usability and Feasibility 

Usability has long been a priority of intervention researchers, with numerous examples of 
practices that researchers have developed in partnership with teachers (e.g., Fuchs & Fuchs, 
1998, Greenwood & Abbott, 2001; Klingner, Boardman, & McMaster, 2013, Vaughn, Hughes, 


Schumm, & Klingner, 1998) to ensure that practices fit within classroom contexts, align with 
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existing instructional programs, and are feasible to implement within the time and resource 
constraints that often exist in schools. Carnine (1997) identified usability, or “the practicality of 
the research-based practices for those who attempt to put them into practice” (p. 514), as a 
marker of “quality” of the educational practice. He suggested that usability of research can be 
improved through increasing relevance, practicality, transportability, and interest. 

Researchers have also attempted to describe facilitators and barriers to teachers’ use of 
research-based practices, in order to find ways to improve usability. For example, Vaughn et al. 
(1998) identified facilitators and barriers to implementing instructional strategies designed to 
facilitate inclusion of students with learning disabilities in general education classrooms. 
Facilitators included the extent to which teachers could adapt or modify the strategy being used; 
students’ acceptance of the strategy; support from researchers (e.g., classroom modeling, 
meeting with researchers); and alignment of the practices with teachers’ personal teaching styles, 
instructional needs of students, and the grade-level curriculum. Barriers included time (e.g., 
competing time demands related to preparing for and implementing the strategies versus time for 
standardized testing and special school events), meeting a wide range of students’ instructional 
needs, and access to materials. 

More recently, Denton, Vaughn, and Fletcher (2003) urged the field to create more 
partnerships between researchers and teachers to bring research-based practices to scale, 
reflecting similar concerns to those raised by Carnine (1997), Vaughn et al. (1998), and others 
from the late 1990s into the mid-2000s (e.g., Boudah, Logan, & Greenwood, 2001; Gersten, 
Vaughn, Deshler, & Schiller, 1997; Greenwood & Abbott, 2001; Klingner & Boardman, 2011). 
They cited the importance of PD in increasing teacher knowledge as well as changing teachers’ 


attitudes and practices, which in turn should lead to improved student outcomes. They described 
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that such PD should include supporting teachers’ understanding of the theory underlying specific 
practices, along with opportunities for practice and immediate feedback to teachers. Further, they 
emphasized the importance of coaching or mentoring by an expert or peers in maintaining 
changes in practice, as well as the importance of teacher characteristics such as self-esteem, 
competence, and flexible thinking in buying into a new practice. 

The persisting challenges related to bridging research and practice recently prompted the 
Institute of Education Sciences (IES) to emphasize usability and feasibility of instructional 
innovations as priorities in development projects (e.g., IES, 2016). In recent requests for grant 
proposals, usability is defined as, “the extent to which the intended user understands or can learn 
how to use the intervention effectively and efficiently, is physically able to use the intervention, 
and is willing to use the intervention” (IES, 2016, p. 52). This language is consistent with 
Carnine (1997). Feasibility is defined as, “the extent to which the intervention can be 
implemented within the requirements and constraints of an authentic education setting” (IES, 
2016, p. 52). A focus on usability and feasibility is central to helping teachers adopt research- 
based approaches to instruction for struggling learners, and so became a focus for our work in 
developing a PD process to support teachers’ use of DBI. Specifically, in this study, we 
examined teachers’ perceptions of facilitators and barriers in using DBI to develop and refine our 
PD process. 

Theoretical Framework 

Our efforts to ensure that DBI was usable and feasible for teachers was informed by 
Desimone’s (2009) essential features of quality PD (content focus, active learning, coherence, 
and collective participation; Desimone & Stuckey, 2014), and led to our theory of change. 


Specifically, we attempted to integrate (a) a content focus, by emphasizing the knowledge and 
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skills teachers needed to implement each DBI step so that intervention was focused on writing 
skills and content and how students learned them; (b) active learning opportunities over an 
extended duration, by providing multiple chances to view models, practice, and apply content 
with feedback during workshops and in the classroom; (c) coherence, by ensuring that DBI 
components align with theory and core standards and are integrated into existing instructional 
routines; and (d) collective participation, by ensuring that teachers have frequent opportunities to 
collaborate with researchers, coaches, and peers. 

Our theory of change was built on a series of assumptions, as follows. First and foremost, 
improving students’ early writing is an important goal. Early literacy and writing skills are 
critical to post-secondary success both in and outside of school (Graham & Perin, 2007), and 
students with disabilities face greater challenges than typically developing peers in this area. For 
example, on the 2011 National Assessment of Educational Progress (NAEP), only 3% of eighth 
graders with disabilities scored at proficient in writing, while 63% performed below basic 
(National Center for Education Statistics, 2012). Fortunately, research has shown that early 
identification and intervention in writing can help prevent long-term negative outcomes for most 
students (Berninger, Nielson, Abbott, Wijsman, & Rasking, 2008). 

Second, to provide effective early intervention, teachers must make timely and 
appropriate decisions for individualizing student instruction, which requires a reliable 
framework—like DBI—that they can implement with fidelity. As illustrated above, researchers 
(e.g., Jung et al., 2017; Stecker at al., 2005) have demonstrated that students’ outcomes can 
improve when teachers use data-based instructional techniques. Third, to implement practices 
such as DBI with fidelity, teachers require knowledge and skills of the practice (Cunningham, 


Perry, Stanovich, & Stanovich, 2004) as well as the belief that they can implement such practices 
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that can lead to student learning (Graham, Harris, Fink, & MacArthur, 2001). One way of 
increasing teacher knowledge, skills, and self-efficacy is through a well-defined PD system. Our 
system provided teachers with Tools (i.e., research-based assessment and intervention tools), 
Learning modules (i.e., face-to-face learning and activities), and Collaborative support (i.e., 
coaching), or ‘TLC’ (Lembke et al., 2018). For additional details regarding our theory of change, 
please see Lembke et al. (2018). 
Purpose 

The purpose of this study was to examine teachers’ perceptions of DBI implementation 
when provided TLC. In doing so, we sought to explore whether teachers’ perceptions might 
reflect whether DBI (i.e., using data to individualize instruction for students with the most 
intensive needs) seemed usable and feasible. DBI has a strong research base (e.g., Jung et al., 
2018; Stecker et al., 2005), but it is used infrequently in practice (e.g., Fuchs et al., 2013). Thus, 
by understanding teachers’ perspectives, we hoped to contribute to the literature on how to 
ensure research-based practices are usable and feasible to teachers. We examined usability and 
feasibility of DBI in the context of strong PD and ongoing coaching provided to teachers, given 
evidence of the importance of such supports (e.g., Denton et al., 2003; Vaughn et al., 1998). Two 
primary research questions guided this work: (a) What did teachers perceive as facilitators and 
challenges to implementing DBI when provided with TLC? and (b) To what extent did they 
perceive implementation of DBI to be usable and feasible when provided with TLC in their 
specific classroom contexts? 

Method 


Setting 
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This study was conducted in a large urban public school district and a mid-sized city 
public school district in two Midwestern states during the 2014-15 academic year. District 1 
served 36,404 K—12 students; 33.6% were White, 37.0% Black, 18.6% Hispanic, 6.8% 
Asian/Pacific Islander, and 4.0% American Indian/Alaskan Native; 64.0% were eligible for 
free/reduced lunch; and 17.8% received special education. District 2 served 17,267 K-12 
students; 62.1% were White, 20.2% Black, 6.1% Hispanic, and 5.1% Asian; 41.2% were eligible 
for free/reduced lunch district-wide; and 9.7% of students received special education. 
Participants 

Participants included special education and English as a Second Language (ESL) 
teachers in the two districts. At the beginning of the study, seven teachers in District 1 and 10 
teachers in District 2 agreed to participate. Two relatively new teachers from District 1 withdrew 
from the study about halfway through the project, citing due process responsibilities taking 
precedence over study activities. A third teacher from District 1 became unresponsive to 
communications toward the end of the study for unknown reasons. Thus, complete data for this 
study were available from four teachers in District 1 and 10 teachers in District 2, for a total of 
14 teachers. 

All participating teachers were female, and all but one identified as White/European 
American (one teacher did not identify her ethnicity). Teachers ranged in age from mid-20s to 
late 40s, and all but one in each district had earned at least a master’s degree. Teachers had a 
wide range of experience, having taught between one to 22 years in their current positions, two 
to 24 years in elementary school, and zero to 24 years in special education. In District 1, all 
participants held certification in elementary (grades 1-6) and special education (with licensure in 


learning disabilities, emotional behavioral disorders, autism spectrum disorders, and other 
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developmental cognitive delays in grades 1-6 or P—12). One held certification in English grades 
7-12, and one in social studies grades 6-8. In District 2, participants primarily held certification 
in elementary, early childhood, and/or special education (mild/moderate disabilities, cross- 
categorical, learning disabilities, and intellectual disabilities) in grades K-6 or K-12. Two also 
held certification in one or more of the following areas: ESL, English, Spanish, French, or adult 
education and literacy. One was a Reading Recovery teacher and instructional coach. In District 
1, all four participants indicated they had received between one and 12 hours of PD or 
coursework in data-based decision making. In District 2, two reported receiving brief trainings in 
CBM. On the whole, the demographics of respondents in this study were representative of 
special educators nationally. In 2014, 85.7% of special educators identified as female and 82.8% 
identified as White (DATA USA, n.d.). 
DBI-TLC 

We developed DBI-TLC as part of a three-year project funded by an IES development 
and innovation grant. In Year 1, we developed DBI-TLC components with extensive input from 
members of the research team, classroom teachers and special education administrators, and 
leading experts in the field of special education. In Year 2 (the focus of this paper), we examined 
the usability and feasibility of DBI with TLC supports. In Year 3, we examined the promise of 
DBI-TLC to improve teacher and student outcomes in a small randomized control trial (see 
McMaster et al., 2020; Poch, Jung, McMaster, & Lembke, 2020). The iterative development of 
DBI-TLC (and its components) is described in detail in Lembke et al. (2018). 

DBI. Across 12 weeks, teachers were asked to implement DBI with two to three of their 
struggling writers in grades 1-3. Using tools developed by the research team, teachers identified 


a primary area of need specific to transcription and/or text generation skills. Next, they 
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developed a Writing Intervention Plan (WIP), incorporating writing practices identified through 
diagnostic evaluation. Writing practices were in the form of short duration lessons (about 5—10 
min each) that addressed critical needs in areas such as handwriting, spelling, and sentence 
construction. Each week, to evaluate students’ response to the intervention provided, teachers 
administered weekly word dictation, picture word, or story prompt CBM probes developed by 
the research team. Although teachers did not implement the same writing lessons, nor did they 
use the same CBM tasks for assessment, given the need to individualize instruction, their use of 
DBI as a framework for their instructional practices was consistent across teachers and was of 
primary interest in this study. Specifically, we taught teachers how to individualize their 
instruction based on student need by creating individualized WIPs and using collected CBM data 
to inform instructional changes. To individualize instruction, we taught teachers to use a 
systematic diagnostic process to consider skill areas in writing in which students were struggling 
(transcription skills including handwriting, spelling, or mechanics; or text generation skills at the 
word, sentence, or passage level). We provided teachers with a writing intervention toolkit with 
suggested intervention ideas (including skills-based mini lessons) that they could choose from 
after considering each student’s strengths and needs. 

In the classroom—with the support of their coach—teachers used weekly CBM 
performance to inform needed changes within their instruction after six to eight weeks. To do so, 
teachers scored, graphed, and analyzed student data using a graphing tool provided by the 
research team. This tool prompted them to enter students’ scores which would then automatically 
populate the student’s graph with both a goal and trend lines. Teachers learned to analyze student 
data visually for level (the position of the data points in relation to the goal line), trend (the 


position of the trend line in relation to the goal line), and variability (the amount of bounce 
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across the data points) in order to make a decision (i.e., keep instruction as is, change instruction, 
or increase the goal). In making a decision, we introduced teachers to a data-based decision 
making flow chart that assisted them in examining the trend line of students’ data, self-checking 
that they implemented the intervention practices with fidelity, and selecting instructional changes 
based on hypothesized students’ needs related to content, focus, and instructional delivery. 

Data Sources 

We primarily relied on qualitative data to answer our research questions regarding 
teachers’ perceptions of facilitators and challenges to DBI implementation and the corresponding 
TLC supports (research question #1). These data came from teachers’ logs (e.g., notes and 
comments), coaches’ logs, focus group conversations, and a coaching survey. See Table 1 for a 
count of qualitative data artifacts by source and district. It is important to note that while each 
participating teacher had an opportunity to contribute data, each teacher is not reflected within 
each individual code. Qualitative data were triangulated with data from quantitative measures 
(e.g., teachers’ logs [e.g., time estimates for DBI activities per child], DBI Knowledge and Skills 
assessment, and the coaching survey) to explore teachers’ perceptions of the usability and 
feasibility of DBI (research question #2). All data sources are described below in the order in 
which participants completed them. 

DBI Knowledge and Skills assessment. Immediately prior to and then following 12 
weeks of DBI implementation, teachers (n = 14) completed a DBI Knowledge and Skills 
assessment, which was designed, piloted, and revised by the research team in Year | of the 
project. Teachers completed the assessment—42 multiple choice questions about CBM, DBI, 
early writing development and intervention, and data-based decision making—via Qualtrics, an 


online survey system. Internal consistency (Cronbach’s alpha) approached the acceptable range 
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of a = .70 (Field, 2009; pre-assessment = .63, post-assessment = .64) after controlling for items 
in which teachers scored 100% at both pre- and post-assessment. This measure was revised 
based on item analyses for later parts of the project, and Cronbach’s alpha improved to .78. 

Teachers’ logs. All participating teachers (n = 14) were asked to maintain logs of time 
spent conducting DBI activities per child receiving DBI, including number of minutes spent 
preparing, administering, and scoring CBM; graphing and examining CBM data; developing 
hypotheses; creating intervention materials; and conducting the entire DBI process. Teachers 
were also encouraged to write any additional notes regarding the feasibility of the DBI process. 
Seven teachers provided additional notes across one to four of the weeks; their notes ranged from 
a single word response indicating state-level testing that may have interfered with 
implementation to a paragraph of 87 words reflecting on the process (e.g., for the students the 
process became routine and for the teacher the process became easier to implement). 

Coaches’ logs. Coaches (n = 9) maintained logs of time spent conducting coaching 
activities, including number of minutes spent in coaching activities, issues addressed in each 
coaching session, the mode of support (e.g., face-to-face, email, phone), the outcome of the 
session, and any additional notes to facilitate weekly team discussions of coaching activities. 
Applicable data from coaches’ logs included the issue addressed during the session, the outcome, 
or additional notes made. The text of these sections ranged from a couple of words (e.g., “ELL 
[English Language Learner] testing”) to paragraphs of at least five sentences detailing what the 
teacher was and was not doing along with questions that the coach could not answer 
independently. 

Focus group protocol. At the end of the DBI implementation period, all teachers were 


invited to participate in focus group sessions to provide feedback on study activities and to 
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reflect on the feasibility of implementing DBI in their classrooms when receiving the TLC 
supports. The principal investigators both led semi-structured conversations within their 
respective districts, and graduate research assistants (three at each site) recorded detailed notes 
on the conversations (complete questions are available from the authors). 

Coaching satisfaction survey. At the end of the study teachers completed a brief survey 
that asked them to (a) evaluate, on a scale of 1 to 10, how integral coaching was to their DBI 
implementation; (b) identify, through two open-ended questions, strengths of the coaching 
process and areas for improvement; (c) indicate, via a multiple choice question, how much 
coaching they felt was needed for teachers to implement DBI successfully; and (d) identify, on a 
scale of 1 to 4, and explain, through two open-ended questions, which components of coaching 
were valuable to their DBI implementation. Teachers’ responses to open-ended questions ranged 
from a couple of words (e.g., “Accessing materials” or “Email meetings”) to paragraphs of about 
four sentences. 

Procedures 

Recruitment. In Fall 2014, teachers from the two districts were recruited to participate in 
the feasibility study with the help of school partners. In District 1, an administrator distributed an 
advertisement describing the study to an email listserv designated for special education teachers; 
interested teachers contacted the Principal Investigator. In District 2, a district administrator 
directly identified and recruited teachers. It is important to note that our recruitment efforts were 
not necessarily aimed at “representativeness,” but rather at identifying teachers who would be 
willing to try DBI in writing. 

Coaching training. In January 2015, the principal investigators at each site implemented 


a 2-hour “Coaches’ Institute” to prepare coaches (advanced doctoral students and project 
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coordinators, all with extensive DBI knowledge and experience working with teachers) to 
conduct coaching activities. This training covered (a) the principles of coaching, (b) appropriate, 
positive coaching behaviors (providing examples and non-examples), (c) how to implement the 
coaching cycle and protocols, and (d) how to differentiate coaching for individual teachers. 

Pre-testing and learning/coaching cycle. Following the Coaches’ Institute, teachers 
received an email invitation to complete the DBI Knowledge and Skills pre-test via Qualtrics. 
Then, teachers attended the Module | workshop, which provided an introduction and overview 
of DBI and CBM. Teachers also learned how to administer, score, and graph CBM. Teachers 
then identified one or two students with intensive writing needs with whom they would 
implement DBI (note that in this phase of the project, student data were not collected). Following 
the Module | workshop, coaching activities commenced to support teachers as they started the 
DBI process with their students. 

Within the next three weeks, teachers received Modules 2 and 3. In Module 2, teachers 
learned to implement research-based early writing instruction, based on the results of a best- 
evidence synthesis of writing instructional practices that members of the research team had 
completed (see McMaster, Kunkel, Shin, Jung, & Lembke, 2018). In Module 3, they learned to 
design individualized WIPs based on students’ needs. Coaches provided individual support to 
teachers in their selection and use of instructional approaches for their students. Module 4 was 
held about 6 to 8 weeks later. In Module 4, teachers learned to use student data to make 
instructional decisions based on the level, trend, and variability of student data. Teachers were 
provided with a decision-making chart that outlined essential questions to ask (e.g., if they had 
implemented practices with fidelity) along with potential intensification strategies if students 


were not responding to instruction. Each module was provided via a full-day training (e.g., from 
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8 am to 3 pm) within teachers’ districts for a cumulative total of 28 instructional hours. Principal 
investigators led each training along with members of the research team. Teachers received 
copies of PowerPoint slides along with accompanying handouts to complete guided practice 
tasks, and at the end of each training, completed performance measures that they reviewed the 
next time they met with their coach. Teachers were asked to continue implementing DBI for a 
total of 12 weeks. Coaches continued to meet weekly or biweekly with teachers in person and 
virtually (via phone or email). Throughout this time, teachers and coaches maintained logs of 
DBI and coaching activities. 

Fidelity. Early in teachers’ implementation of DBI, coaches observed teachers’ fidelity 
of CBM administration and writing intervention implementation using modified versions of the 
Accuracy of Implementation Rating Scales (AIRS), originally created by Fuchs, Deno, and 
Mirkin (1984). Each protocol (AIRS-CBM and AIRS-Writing Instruction) consisted of a detailed 
checklist of assessment and intervention steps. Fidelity was recorded as the number of steps 
observed over total number of applicable steps. Mean fidelity of CBM administration (as 
measured by AIRS-CBM), completed for 10 of the 14 teachers, was 95.35% (SD = 10.57, range 
= 66.67% to 100%). Mean fidelity of writing intervention (as measured by AIRS-Writing 
Instruction), completed for 9 of the 14 teachers, was 94.72% (SD = 10.19, range = 87.50% to 
100%). Note that the AIRS tools were under development at this point in the project; we realized 
that the AIRS-Writing Instruction tool did not seem sufficiently detailed to capture teacher 
differences in implementation, and we subsequently revised this tool for use in later stages of the 
project. 

Following Module 3, coaches were asked to audio record one of their coaching sessions 


for a fidelity check; seven of the nine coaches whose teachers provided complete data for the 
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study successfully did so. The lead author reviewed the audio files using the Fidelity of Coaching 
form which contained 16 items divided across nine core areas: (a) preparation (four items), (b) 
celebrate and commiserate (one item), (c) summary of previous meeting (if applicable; one 
item), (d) objectives for the meeting (one item), (e) review of performance measures (if 
applicable; one item), (f) review/discuss DBI process (if applicable; two items); (g) 
review/discuss DBI observation (if applicable; two items), (h) review/discuss student data and 
intervention plans (if applicable; two items), and (i) next steps (two items). The observer noted 
whether each component was observed or not observed. Fidelity was recorded as the number of 
components observed over the total number of possible components. The second author then 
reviewed three of the seven audio files (selected at random). Agreement ranged from 75% to 
87.5%; discrepancies were discussed and resolved. Fidelity of coaching ranged from 66.67% to 
100%, with a mean of 87.99%. Lower fidelity resulted from coaches failing to complete one or 
more of the following agenda items: celebrating and commiserating with the teacher (n = 1), 
summarizing the previous coaching session (n = 3), stating goals and objectives for the current 
meeting (7 = 1), setting goals for the next session (m = 1), and setting the next meeting time (1 = 
1). Overall, these results indicate that coaching was implemented as intended, with only minor 
protocol deviations. 

Focus groups and post-testing. In May 2015, teachers were invited to participate in the 
focus group sessions. Following the focus group sessions, teachers were provided time to 
complete the DBI Knowledge and Skills post-assessment and coaching satisfaction survey. 
Research Design and Data Analysis 

To determine the usability and feasibility of DBI-TLC, qualitative and quantitative data 


were analyzed within a convergent triangulation mixed methods design, which allows for the 
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synthesis of complementary qualitative and quantitative results to develop a more complete 
understanding of a phenomenon or research problem (Creswell & Plano Clark, 2011). The 
convergent design contains four primary steps: (a) qualitative and quantitative data are collected 
concurrently yet separately, (b) the data are analyzed independently of each other using 
appropriate qualitative and quantitative procedures, (c) results from the qualitative and 
quantitative data are merged, and (d) results are interpreted to determine the ways in which the 
qualitative and quantitative results converge (Creswell & Plano Clark, 2011). 

Qualitative analysis. The first and second authors entered the qualitative data—notes 
from teachers’ logs, coaches’ logs, and the focus group conversations, as well as open-ended 
question responses from the coaching survey—from their respective districts into Microsoft 
Excel (2007) spreadsheets, and then coded the data using a constant comparative approach 
(Corbin & Strauss, 2008). Below, we describe this approach. 

Positionality. An integral part of qualitative data collection and analysis includes 
positioning oneself within the research and objectively stating one’s biases and theoretical 
assumptions to promote credibility (Brantlinger, Jimenez, Klingner, Pugach, & Richardson, 
2005). The researchers who collected and analyzed the data shared a strong orientation toward 
the use of evidence-based instructional practices to promote students’ academic achievement, 
and the use of DBI (using reliable and valid progress monitoring data) to individualize and 
intensify instruction for students with the most significant academic needs. Further, we believed 
that teachers just learning to implement DBI in early writing would benefit from high-quality 
PD, tools to implement assessment and intervention procedures, and ongoing support to 


implement DBI with fidelity. 
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Additionally, research team members had strong backgrounds in the development, 
evaluation, and use of CBM and research-based interventions. We shared the belief that students 
with or at-risk for disabilities can learn and that they should be provided with the necessary 
supports to be successful and to make measurable growth. We also believed that special 
educators could be powerful change agents in students’ lives, particularly when given the 
knowledge and skills to affect such change. Our beliefs and assumptions not only influenced our 
development and implementation of DBI-TLC, but also the types of data that we collected and 
the way that we coded, analyzed, and interpreted these data. 

Coding process. First, the two coders (the first and second authors) read the original data 
and identified potential categories for coding. This was an iterative process lasting approximately 
12 months because the data were also being used to inform further development of DBI-TLC for 
a small randomized control pilot study the following year. The lead author brought some initial 
ideas to the first coding meeting (e.g., teacher challenges and coach’s response, tied to coaching 
standards). However, in discussion, the first two authors decided to initially code by category 
(e.g., descriptive, revision, both, neither). We then started to pull out DBI components within the 
data. Later, the authors experimented with the concepts of barrier/challenge and with using the 
coaching principles the team had identified to code the data. These discussions and codes led to 
considerations of a possible hierarchical nature to the data where Level 1 codes were identified 
(i.e., Facilitator, Barrier/Challenge, Sustainability/Feasibility) and later slightly revised for 
wording. Time with the data and consistent communication of ideas led to Level 2 codes aligning 
with the components of DBI-TLC, and in vivo coding for Level 3 codes. Table 2 provides an 
evolution of the coding structure with the application of one data artifact as an example. In this 


table we take one data artifact through the stages of coding (column 1) outlined above, 
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identifying possible codes (column 2), how the example artifact was coded within the identified 
stage (column 3), and problematic aspects of the coding stage (column 4). 

Level 1 codes focused on feasibility in the form of what teachers perceived to be 
facilitators and challenges. Facilitators were coded as either actual facilitators—features of DBI- 
TLC that, from the teachers’ perspectives, fostered or enabled their DBI implementation during 
the project period, or their participation in DBI-TLC, or suggested facilitators—features that, if 
incorporated into DBI-TLC, teachers perceived would have facilitated their implementation of 
DBI or their participation in DBI-TLC. Challenges were features that teachers perceived to make 
the implementation of DBI difficult or hampered their participation in DBI-TLC. 

Level 2 codes were aligned with the primary components of DBI-TLC: DBI (i.e., the DBI 
process [including assessment, intervention, and decision making] and/or the data collected via 
DBD), tools (1.e., any of the products developed for and used in DBI), /earning modules (i.e., 
anything related to the learning opportunities provided through the learning modules), and 
coaching (i.e., anything related to coaching behaviors, actions, requests, or supports, including 
both face-to-face and virtual [e.g., email] interactions between coach and teacher). 

Level 3 codes were more specific descriptions of the data (e.g., a specific feature of DBI- 
TLC that teachers perceived to be a facilitator or challenge to their implementation or 
participation). Counts of Level 3 codes are summarized in the coding matrix in Figure 1. 

This coding structure was applied in several “rounds” of coding, during which we both 
individually coded our respective district’s data, compared notes and came to consensus on how 
to refine the coding structure and language, and recoded the same data based on our discussions. 
Once we agreed on a complete set of codes and finalized coding in our respective datasets, we 


exchanged datasets and coded each other’s (blinded to the other’s codes). We discussed 
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disagreements in codes and further refined the coding structure until we reached complete 
agreement on all codes, and finally combined the two datasets. Throughout this process, we 
developed and refined definitions of each code, and used these definitions to constantly ensure 
consistency in our coding process. The finalized code book is available from the authors. 

Promoting trustworthiness and credibility. We took several steps to promote 
trustworthiness and credibility throughout data analysis to validate our coding process and 
structure. First, the coders met, at a minimum, every other week; during iterative rounds of data 
analysis (i.e., when actively coding), we met weekly. These meetings, along with frequent email 
exchanges, allowed for flexibility and reflexivity in our analytical thinking. Following each 
meeting, the first author updated a list of steps taken in data analysis; she also maintained 
handwritten notes taken during each meeting. These documents provided an audit trail. 
Additionally, triangulation of data was completed across sites and across data sources (teachers’ 
and coaches’ logs, focus group notes, and coaching satisfaction survey data); greater detail is 
provided later in the manuscript. 

We also conducted member checks of the coding structure—first with our research team 
and then with participating teachers. After the primary coders agreed on all codes and 
definitions, we presented our coding structure, definitions, and examples to the rest of the 
research team and answered their questions. We created two forms containing items we had 
coded (with codes removed) and asked three members of each team to complete one of these 
forms to assess the ease and accuracy of the coding structure. Each form contained a unique, 
representative sample of a portion of the coded items, with an equal number of items per form (n 
= 17). Three open-ended social validity questions were also included on each form: (a) How easy 


was it to apply the codes? (b) Do the codes fit with your interpretation of teacher perceptions? 
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and (c) Is the wording of the codes clear and consistent? The purpose of this activity was to 
ensure that our coding structure made sense to others who were familiar with the study and the 
data, as well as to ensure that the terms and definitions that we used were clear and consistent. 

After the team members completed this exercise, we compared their codes to our own. 
Each data artifact could receive up to three points—one for each correctly labeled code—for a 
possible 51 points. Percent agreement between the team members’ codes and the original codes 
ranged from 66.67% to 82.35%. The first and second authors (original coders) examined codes 
on which there was disagreement to determine whether the codes were incorrect or whether 
definitions needed to be clarified. The result was a confirmation of the existing coding labels on 
all but a couple of items; those items were re-labeled. During this same time, the original coders 
re-sorted the coding spreadsheet to ensure that similar items were coded (and thus grouped) 
appropriately and to identify and re-evaluate any codes that had only been used once; this 
analysis led to slight modifications to the coding structure (e.g., a code was collapsed with 
another item or relabeled to more appropriately reflect the data). We also reviewed and revised 
definitions for clarity. 

As a final step, the first and second authors completed member checking with two 
teachers from District 1 and four teachers from District 2 during Summer 2016. These meetings 
lasted between 60 and 90 min and followed a standard agenda in which we (a) reminded teachers 
of the study’s purpose, (b) described the coding process, (c) walked teachers through the coding 
structure, and (d) asked them to review the coded items. We asked if the codes were clear, if they 
agreed or disagreed with the structure or specific codes, and if we missed anything. In both sites, 
teachers reported that the coding structure was accurate and reflective of their experiences in the 


project, and that the wording of the codes was clear and consistent. 
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Quantitative analysis. Quantitative data analysis included (a) examining teachers’ pre- 
to posttest growth on the DBI Knowledge and Skills assessment using a paired-samples t-test, (b) 
summarizing the amount of time that teachers reported spending on DBI activities per child each 
week (using means and SDs for number of minutes spent on each activity), and (c) summarizing 
teachers’ quantitative ratings on the coaching satisfaction survey (reporting means and ranges for 
each relevant question). 

Triangulation and validation of findings. Results from quantitative analyses were 
triangulated with qualitative results to determine the extent to which findings converged across 
data sources. Specifically, teachers’ comments about knowledge and skills gained from the PD 
that was provided were compared to quantitative data from the Knowledge and Skills assessment 
to determine whether perceived growth related to actual growth. Teachers’ comments about the 
time consumed by various DBI activities across time were compared to quantitative data from 
teachers’ logs. Teachers’ comments about the value of coaching were compared to their ratings 
on the coaching satisfaction survey. All of the findings were validated through the member 
checking process described above. To further validate the reporting of the results, we attempted 
to adhere to quality indicators for qualitative (Brantlinger et al., 2005) and mixed methods 
(Leech & Onwuegbuzie, 2010) research. These indicators were examined point-by-point across 
the manuscript by the third author. 

Results 

The purpose of this study was to examine teachers’ perceptions of DBI implementation 

when provided TLC through the lens of usability and feasibility as they worked with first 


through third graders at risk or with disabilities and identified with intensive writing needs. 
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Qualitative data served as our primary data source, and quantitative data as a secondary source to 
triangulate results. 

Below we provide more details and examples of teachers’ perceived actual and suggested 
facilitators as well as challenges, in order of most frequently occurring codes. See Figure 1 for 
counts of codes. Note that some of the teachers’ statements are offset by quotation marks; these 
statements are as close to verbatim as possible but may not be exactly what the teachers said, as 
data were drawn from field notes and coaches’ and teachers’ written statements rather than 
transcribed audio recordings. 

Facilitators of Teachers’ DBI Implementation 

Teachers consistently indicated perceived actual facilitators—features of DBI-TLC that, 
from the teachers’ perspectives, contributed to their use of DBI and/or their participation in DBI- 
TLC. The majority of actual facilitators related to coaching (n = 66). These facilitators were 
explained by 14 Level 3 codes, which primarily described how coaches assisted and supported 
teachers’ use of DBI. For example, teachers reported the general support (n = 11) that their 
coach provided as facilitating their implementation of DBI: “[Coach] was always available when 
I needed her. She was very supportive and checked in with me regularly.” Others identified that 
coaches provided accountability (n = 8), such as when a teacher from District 2 reported that her 
coach facilitated her use of DBI by “reminding me of the process; holding me accountable.” 
Another teacher identified the help with implementation (n = 8) that she received: “My coach 
helped me with scoring, accessing materials, graphing data and planning intervention.” 

Actual facilitators related to DBI (n = 31) were explained by eight Level 3 codes. Most 
responses reflected the value (n = 9) that teachers attributed to DBI. For example, during the 


spring focus group, a teacher from District 1 indicated that being engaged in the project 
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“Reminded me of being back in college. [It] brought me back to the whole reason of what we do: 
We re-looked at what best practice is, which our time and caseloads don’t allow. [I] felt like all 
data was valuable and informed practice.” 

Actual facilitators related to tools (n = 19) were explained by four Level 3 codes and 
again emphasized value (n = 6). One teacher commented that the “DBDM [data-based decision 
making] rubric was helpful. It [acts] as a good reminder of all the different ways you can make a 
change.” 

Actual facilitators related to the learning modules (n = 4) were explained by two Level 3 
codes, and reflected timing (n = 3) and the content of the learning modules (n = 1). Here, timing 
was specific to when the learning modules occurred, as when a teacher indicated, “[I] enjoyed 
having it broken down into different sessions: [I] left each time feeling like you learned 
something useful, things that I can implement right now.” This teacher’s statement also reflects 
the content of the learning modules as teachers received training on a different aspect of DBI 
during each module. 

Teachers also suggested facilitators, or supports, that they perceived would have 
improved their implementation of DBI and/or their participation in DBI-TLC. The majority of 
their suggestions related to coaching (n = 35) and tools (n = 25). Suggested facilitators specific to 
coaching were explained by six Level 3 codes, where many teachers recommended additional in- 
class supports (n = 18; e.g., “have coaches come during your lessons more than once; make sure 
we’re still doing everything we’re supposed to, watch kids who might be having trouble’). 


Suggested facilitators specific to tools emphasized structure (n = 9; e.g., “Yes, teachers want you 


to tell them what to do” —+teferring to teachers’ desire for a pre-made, researcher-developed WIP 


that they could follow rather than having to create their own). 
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Suggested facilitators specific to DBI (n = 9) and the learning modules (n = 5) both 
emphasized timing. Timing for DBI (n = 5) was specific to the best time within the school year 
to start DBI (e.g., “starting in the fall will be helpful—when schedules do change in the spring it 
will be easier to get back into it if you’ve already been doing it all along”) whereas timing for the 
learning modules (n = 2) were suggestions about when to provide a learning module or a 
reflection about when a learning module occurred (e.g., “Maybe a combo of summer and during 
the year training”). Suggested facilitators for the learning modules equally emphasized the 
learning curve (n = 2) that teachers experienced, or the idea that the DBI process is difficult at 
first and teachers need ample time to learn and practice the procedures. This idea was captured 
when a teacher stated during the focus group, “[you] need two weeks or more to play with mini- 
lessons and experiment; I wasn’t really doing it until a month in. It gets easier.” 

Overall, the structure of DBI-TLC promoted routine, ongoing practice, which was 
reinforced by coaches’ help and feedback, as well as a sense of accountability, which prompted 
them to engage in the necessary practice of using data to inform ongoing instruction. Teachers 
also cited that when DBI and the tools provided aligned well with their existing instructional 
programming, their students’ needs, and their own values related to using data and implementing 
research-based interventions, they were more likely to implement and sustain DBI. However, 
teachers would have liked additional in-class supports, such as more face-to-face time and 
guidance during instruction to ensure that they were implementing DBI correctly, along with 
more collaboration time with peers. 

Challenges 
Teachers also identified challenges that they perceived impeded their ability to 


implement DBI and/or participate in DBI-TLC. Most of the identified challenges were specific to 
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the tools (n = 40) and the DBI process (n = 21). Only a few data artifacts were specific to 
coaching (n = 6) and no challenges were identified related to the learning modules. 

The greatest challenge that teachers noted about the tools was accessibility (n = 20). This 
challenge was reflected in statements such as “some materials printed [from the Google drive] 
earlier but won’t now...maybe the format was changed?” and “Finding materials in Google 
docs.” This finding is consistent with earlier work noting that the inaccessibility of tools needed 
to carry out an intervention can diminish the extent to which teachers value the intervention and 
implement it with fidelity (e.g., Vaughn et al., 1998). 

The greatest challenges that teachers noted related to both the DBI process and coaching 
were external conflicts (n = 13 and n = 4 respectively) that interfered with their ability to engage 
in DBI-TLC, such as testing, student absences, and the time of the year. As one teacher reflected 
during the focus group when asked about the greatest challenges of implementing DBI, “Time of 
year; [it’s] impossible to do anything with [state assessments], district testing, [and] IEP season.” 
Another teacher remarked, “Sometimes keeping the time that you set with your coach was hard 
for me. Mandatory meetings and testing schedules change.” The struggle was equally evident on 
the coach’s end; one coach wrote in her coaching log, “[teacher participant] is not following up 
on first grade student in terms of intervention—she [teacher participant] and [1* grade teacher at 
the school] have not had time to sit down and go through WIP she planned for student, so she 
also has not PM [progress monitored] student.” 

Overall, the accessibility of materials was the greatest challenge that teachers perceived 
they experienced in this study specific to tools. For the DBI process and coaching, the greatest 
perceived challenges were external conflicts such as those stemming from state testing, the time 


of year (beginning in Spring), and regular classroom demands; these external conflicts impacted 
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not only teacher implementation and use of DBI, but also coaches in trying to ensure teachers 
were following through with study expectations. 
Usability and Feasibility 

Qualitative findings described above were triangulated with quantitative findings to 
answer questions regarding teachers’ perceived usability and feasibility of DBI in the context of 
TLC. Below, we report quantitative results that provide evidence of teachers’ perceived 
usability, the extent to which the practice can be learned and used, and feasibility, or the extent to 
which the practice can be implemented within an authentic setting (IES, 2016), and connect these 
findings with qualitative results. 

DBI Knowledge and Skills. On the DBI Knowledge and Skills assessment, the average 
score at pretest was 28.45 items correct out of 42 (SD = 4.68). At posttest, the average score was 
36.27 (SD = 2.69). This pre- to posttest change was statistically significant (t{10] =— 6.659, p < 
.001; see also Lembke et al., 2018). These results provide evidence that teachers learned critical 
components of DBI in early writing and these results converge with qualitative evidence that 
teachers felt that they learned from participating in DBI-TLC (e.g., prior knowledge, usability, 
and value) and improved their skills over time (e.g., learning curve). 

Time to implement DBI activities. Data from teachers’ time logs for each student 
(collecting, scoring, and graphing CBM data; preparing instructional plans; making instructional 
decisions) are presented in Table 3. These descriptive data revealed that time spent on each 
activity decreased substantially across 12 weeks, with overall time spent on DBI-related tasks at 
53 minutes in Week 1, and 31 minutes by Week 12 (note that this information does not include 
actual instructional time). Time spent on CBM-related activities decreased steadily, but time 


spent preparing instructional materials fluctuated and remained relatively high. These results 
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suggest that teachers were able to implement the assessment components of DBI more efficiently 
over time; these results also converge with qualitative evidence that teachers felt that DBI 
generally became more manageable as they gained experience. The finding that time spent 
preparing instructional materials did not steadily decrease is consistent with qualitative 
comments in which teachers’ perceived that material preparation was one of the more 
cumbersome aspects of DBI. This finding might also reflect that teachers were making 
instructional adjustments and thus needed to continue preparing new materials. 

Satisfaction with coaching. Teachers who completed the coaching survey (n = 13) rated 
coaching as highly integral to their DBI implementation (M = 8.77 out of 10, SD = 1.42); 
suggested that coaching should be implemented face-to-face on at least a monthly basis (n = 4), 
with some teachers indicating more frequent coaching is needed (e.g., every other week, n = 5) 
or that it depends on the teacher (n = 2); and that all aspects of coaching are valuable, 
particularly problem solving about student data and intervention (M = 3.69 on a 4-point scale, SD 
= .63 for both). These results converge with qualitative data indicating that teachers perceived 
coaching to generally facilitate DBI implementation, particularly by holding them accountable, 
providing general support, and offering meetings and check-in communications. 

Discussion 

In this study we explored teachers’ perceptions of facilitators and challenges to 
implementing DBI with TLC supports to assess DBI’s usability and feasibility. Our results 
support three core findings that are both consistent with and extend the current literature on DBI. 

First, teachers were willing to work though challenges to using new interventions when 
provided with supports to facilitate their understanding and when those facilitators and supports 


aligned with current instructional practices. The facilitators that teachers noted in this study are 
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consistent with the facilitators identified by Vaughn et al. (1998; e.g., ability to adapt/modify the 
intervention, access to materials and classroom/research partners, in-class supports, instructional 
needs of students). For our teachers, being able to align DBI with students’ individual needs as 
well as the curriculum, and having a support network of both researchers and fellow teachers 
who were also implementing DBI, were critical. Our teachers even suggested that more of these 
supports would have further enhanced their instructional practices in using DBI. Although the 
specificity and intensity of facilitators may vary by teacher and school, the facilitators necessary 
for supporting teachers’ use of new interventions like DBI (e.g., ability to adapt or modify 
strategies and materials, fit with curriculum and student needs, support from researchers and 
fellow teachers involved in implementation) are potentially most effective when they work in 
conjunction with each other. 

Second, teachers’ DBI knowledge and skills and fidelity of implementation improved 
when they were provided effective PD that fostered the development of these skills. And, third, 
DBI was feasible to implement over time with ongoing support and materials. Both Carnine 
(1997) and IES (2016) defined usability as the practicality of an intervention, or the extent to 
which a user—such as a teacher—understands and can learn how to use an intervention 
effectively and efficiently. Feasibility refers to the extent to which the intervention can be 
implemented and used within the classroom setting (IES, 2016). Fostering both usability and 
feasibility of DBI may necessitate supporting both teachers’ understanding of DBI when 
provided TLC and teachers’ understanding of the theory underlying specific practices, along 
with opportunities for practice and immediate feedback to teachers (e.g., Carnine, 1997; Denton 


et al., 2003; Vaughn et al., 1998). Such a foundation may also help foster a shift for teachers 
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from knowledge acquisition to knowledge transformation, in which teachers activate their new 
learning to effect change in classroom practice. 

Thus, our results emphasize the importance of various supports necessary for building 
teachers’ use of DBI (e.g., supports for understanding/learning, instructional/curricular 
alignment, training, and materials). Indeed, the current literature points to the importance of 
ongoing supports by experts or peers (or even technology)—which we find as well, but we also 
find that teachers’ knowledge and skills as well as access to assessment and intervention 
materials are critical. These latter supports are new to the DBI literature at an individual teacher 
level and speak to the importance of identifying teacher-level factors that can help sustain 
teacher use of data-based approaches to supporting struggling learners. 

Limitations 

Findings of this study should be interpreted within the context of the following 
limitations. First, the data that we report are from a small group of teachers who opted to 
participate (and persisted throughout the study), which might be very different from the 
perspectives of teachers who never joined the study or who discontinued participation. 
Additional information from teachers who opted out of the study at different points could 
provide greater insight about the feasibility of DBI-TLC, as well as the characteristics of teachers 
who are likely to find this process usable and feasible to implement. A second limitation is that 
most of the qualitative data in this paper were derived from teachers’ responses to specific 
questions that we asked; the focus and proportions of types of facilitators and challenges might 
have been different if we had asked different questions or had the questions been more open- 
ended. Furthermore, although the suggested facilitators help to expand upon the current research 


surrounding facilitators and barriers to teachers’ use of research-based practices, the suggested 
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facilitators presented here may not be generalizable outside of this study. It is possible that 
teachers in other contexts would identify suggested facilitators more specific to their own 
situations. Additionally, we did not audio record and then transcribe the focus group meeting, 
relying instead on notes taken by advanced graduate assistants, which reduced an already small 
corpus of data and potentially involved a layer of interpretation prior to coding. Third, teachers 
provided a self-report of the time it took them to implement DBI. Thus, it is possible that 
teachers over- or underestimated the amount of time that it took them to complete these 
activities. Fourth, our fidelity measures were under development during this iteration of the 
project. While they required revision prior to completing a small follow-up randomized control 
study, our fidelity measures were adapted from current measures with strong research support. 
Fifth, because we were only interested in examining usability and feasibility, our data cannot be 
used to determine whether the improvements in teacher beliefs that we noted, contributed to 
changes in teacher instruction or student performance. 
Implications for Research and Practice 

Although this study provides evidence of teachers’ perceived usability and feasibility of 
DBI (when provided TLC), along with the importance of coaching in facilitating teachers’ use of 
DBI, additional work is needed to determine the sustainability of DBI-TLC, including whether or 
not teachers maintain their use of DBI once supports are withdrawn and whether DBI is useable 
and feasible when teachers are not provided with supports. Our teachers were not tracked over 
time to determine whether the PD and supports they received were sufficient for maintaining 
DBI beyond the project period. We also recognize that teachers and schools do not always have 
access to the types of supports that we provided in this study, which could make sustained 


implementation difficult. However, it is possible that establishing teachers’ baseline performance 


USABILITY AND FEASIBILITY OF DBI 35 


and familiarity with such tasks as administering and scoring CBM and making data-based 
decisions might provide for varying levels of coaching based on teacher need. 

Throughout the process of completing this study, we learned, like other researchers 
before us (e.g., Vaughn et al., 1993), the importance of extensive teacher input as a necessary 
and integral component for developing and validating complex instructional frameworks (Chard, 
2004; Denton et al., 2003; Klingner et al., 2013) such as DBI. Indeed, building bridges between 
research and practice requires considerable time and investment, along with the balancing of 
multiple agendas (Vaughn et al., 1998), a point that has not been lost on researchers within the 
last couple of decades. Building close relationships with the teachers in this study allowed for 
careful critique and re-evaluation of the PD system that we developed. The majority of data 
sources, both qualitative and quantitative, were provided through teacher input: teachers’ logs, 
focus group conversations, a coaching survey, and the teachers’ Knowledge and Skills 
assessment. Moreover, the coaches’ logs documented coaches’ interactions with teachers, 
including teachers’ feedback, reflection, challenges, and successes, that were discussed at weekly 
research team meetings. Though our tools and procedures were grounded extensively in 
evidence-based instructional practices and established theory surrounding PD, coaching, and 
writing instruction, we learned valuable lessons (e.g., regarding managing instructional routines 
for students of diverse learning abilities, teachers’ struggles with technology, and the desire for 
more in-class supports tailored to teacher need) that might not have been realized without teacher 
feedback and collaboration. Findings of this study support the notion that teachers’ adoption and 
use of evidence-based practices, such as DBI, is most likely to happen through sustained, 


collaborative relationships between researchers and practitioners. 
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Table 1 
Number of Qualitative Data Artifacts by Source and District 
Data Source District 1 District 2 Artifacts Totals by Data 
Missing Source 
District 
Identifier 
Coaches’ Logs 11 26 0 oy 
Coaching Survey 16 38 1 55 
Teachers’ Logs 4 11 0 15 
Focus Group 4] 74 0 115 


Artifacts Missing 

Data Source 

Identifier 1 0 1 2 
Totals by District fie 149 2 224 


Note. Values exclude 14 items (e.g., two coaches’ log artifacts from District 1 and six focus 
group artifacts from both District 1 and 2) coded as “other” and excluded from analysis. 
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Table 2 
Evolution of the Coding Structure 


Example Data Artifact Taken Through the Coding Process 
[Teacher] followed the lesson plan well, but we both noted at the end of the lesson that there was not sufficient 
detail provided for teaching the lesson (Sentence Combining using "and") 


Stage Coding Example Problematic Aspects 

1 — Initial = Teacher challenges and N/A Intended for brainstorming and 
Thoughts coach’s response examining the data to develop an initial 
= Tied to coaching standards impression of the data; labeling did not 


fit all data, making coding decisions 
seem forced. 


2= = Descriptive Coded as Focused solely on informing pilot study 
Categorical = Revision Revision in Year 3 rather than consider feasibility 
Labels = Both of DBI implementation; focus was on 

= Neither logistics which was a step toward 


thinking about feasibility based on what 
potentially needed to be changed. 


3 — DBI = Sustainability Coded as This was an initial attempt to align the 
Components = Coaching Intervention data artifact to the DBI components 
= DBI-TLC central to our study. This coding stage 
= DBDM also allowed for continuing to refine and 
= Intervention match members of the research team to 
= Training targeted aspects for revision. 


= Google Drive 
= Graphing/Graphing Tool 
= Toolkits/Progress Monitoring 


»= CBM 
= Collaboration 
4 — Leveled = Level 1 (feasibility/ Coded as While coding for DBI components a 
Coding sustainability, facilitator, Barrier/ series of notes and expanded notes were 
barrier/challenge) Challenge maintained by the coders which 
= Level 2 (more specific (L1) and informed a potential leveled coding 
category) Missing system, but this analysis did not quite 
= Level 3 (more specific than lesson consistently capture the data we had 
Level 2) component collected. 
(L2) 
5 — Coaching = Barrier/Challenge Coded as How the coding structure applied to 
Principles = Coaching Principles Instructional particular data artifacts was unclear 
= Building Mastery Practice leading coders to question the utility of 
= Teacher Oriented these codes. 
= Sustainability 
= Instructional Practice 
= Measurement 
6- =" Level 1 (actual facilitator, Coded as More complex, but a better 
Hierarchical suggested facilitator [formerly Challenge representation of the data which 
Coding sustainability/feasibility ], (L1), Tools/ combines elements of previous coding 
barrier/challenge) Materials stages. 
= Level 2 (DBI, learning (L2), and 
modules, TLC) Accessibility 
=" Level 3 (in vivo codes) (L3) 


Note. N/A = not applicable; DBI = data-based instruction; DBI-TLC = data-based instruction—tools, learning 
modules, and collaboration; DBDM = data-based decision making; CBM = curriculum-based measurement; L1 = 
level 1; L2 = level 2; L3 = level 3; TLC = tools, learning modules, and collaboration. 
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Table 3 
Number of Minutes Spent on DBI Activities per Week per Child 
Preparing Graphing Preparing Total Time 
CBM Administer- Scoring CBM Examining Developing Intervention Implementing 
Materials ing CBM CBM data CBM data Hypotheses Materials DBI 

Week 1 

M 14.40 13.64 19.43 10.67 10.00 10.00 17.50 52.80 

SD 15.54 10.63 16.46 11.39 - - 9.57 41.60 

n 25 22 21 9 3 1 4 25 
Week 2 

M 7.50 9.09 15.50 6.65 11.50 9.50 35.00 53.88 

SD 8.30 8.65 9.72 4.58 6.26 3.69 45.03 39.67 

n 22; 23 20 17 10 10 10 25 
Week 3 

M 3.81 5.78 9.33 5.25 6.67 7.27 29.23 39.50 

SD 2.32 3.56 3.20 5.37 2.50 2.61 12.89 22.50 

n 21 18 15 16 9 11 13 24 
Week 4 

M 7.62 5.05 8.67 3.17 6.00 - 32.19 40.65 

SD 3.86 3.44 3.52 1.95 4.08 - 49.11 47.28 

n 13 21 15 12 7 0 16 23 
Week 5 

M 5.18 6.23 8.76 3.19 6.43 9.67 32.27 34.29 

SD 5.15 3.94 3.68 2.61 3.41 13.28 27.78 36.46 

n 11 22 17 16 7 3 11 24 
Week 6 

M 5.40 4.35 5.57 2.94 6.38 10.00 24.09 27.50 

SD 2.72 1.56 3.08 1.92 3.11 25.18 27.36 

n 10 23 21 18 8 3 11 24 
Week 7 

M 5.56 4.45 6.70 3.52 6.25 8.33 18.00 31.29 

SD 3.91 1.61 2.67 2.17 2.32 2.89 38.35 31.46 

n 18 20 23 23 8 3 14 24 
Week 8 

M 5.87 4.65 5.74 2.62 4.78 15.42 27.26 

SD 4.97 1.57 3.08 1.66 2.28 1.16 14.69 20.86 

n 15 20 23 21 9 12 23 
Week 9 

M 4.05 4.76 6.86 4.70 6.22 10.83 23.00 34.82 

SD 2.72 1.48 3.25 9.17 3.87 9.70 34.50 37.55 

n 19 21 21 20 9 6 10 22 
Week 10 

M 3.25 4.39 6.96 4.25 8.44 10.00 66.25 46.00 

SD 1.84 1.56 3.24 TAI 5.88 5.48 57.74 45.90 

n 16 23 23 20 9 6 8 24 
Week 11 

M 4.63 4.35 6.65 3.15 7.36 8.33 43.18 46.77 

SD 2.75 1.66 3.08 1.81 4.29 5.16 72.95 50.33 

n 19 20 20 20 14 6 11 22 
Week 12 

M 3.25 4.25 6.74 3.05 4.71 15.00 26.11 30.70 

SD 1.84 1.65 3.14 1.81 2.63 13.23 17.28 23.90 

n 16 20 19 19 7 3 9 23 
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Note. CBM = curriculum-based measurement; DBI = data-based instruction. 
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Figure 1. Coding Matrix (values in parentheses denote total counts). 


Level 1—» 


Actual Facilitator (n = 120) 


Suggested Facilitator (n = 74) 


Challenge (n = 67) 


Level 2 


Answering questions (3) 
Coach knowledge (1) 
Feedback (5) 

Decision making (4) 

General support (11) 

Help with implementation (8) 
Meetings and check-ins (7) 
Motivation/encouragement (3) 
Problem solving (4) 
Sustainability (4) 

Timing (2) 

Value (2) 


In-class supports (18) 

Individualized options for coaching (8) 
Teacher responsibility (1) 

Timing (2) 


DBI = Alignment (5) = Alignment (1) = External conflicts (13) 
(n= 61) = External supports (1) = External supports (2) = Teacher responsibility (1) 
= Collaboration (3) = Timing (5) = Learning curve (6) 
= Content of learning module (1) = Learning curve (1) = Timing (1) 
= Learning curve (6) 
= Prior knowledge (3) 
= Structure (2) 
= Usability (1) 
= Value (9) 
Tools = Accountability (3) = Accountability (1) = Accessibility (20) 
(n = 84) = Alignment with instructional program = Alignment with student needs (6) = Alignment (4) 
and student needs (5) = Management (6) = Learning curve (3) 
= Usability (5) = Structure (9) = Management (8) 
= Value (6) * Usability (3) = Usability (5) 
Learning = Content of learning modules (1) = Learning curve (2) 
(Workshops) = Timing (3) = Teacher responsibility (1) 
(n = 9) = Timing (2) 
Coaching = Accessing or creating materials (4) = Coach knowledge (3) = External conflicts (4) 
(n = 107) = Accountability (8) = Decision making (3) = Teacher non-responsiveness (2) 


